Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update triton to allocate more cuda memory to pool size #237

Merged
merged 1 commit into from
Dec 1, 2022
Merged

update triton to allocate more cuda memory to pool size #237

merged 1 commit into from
Dec 1, 2022

Conversation

jperez999
Copy link
Collaborator

This will help stabilize the triton runtime fil op tests. It allows triton to allocate ~500MB instead of the default 64MB. This helps when loading models specifically FIL backend models.

@jperez999 jperez999 marked this pull request as ready for review November 29, 2022 16:17
@jperez999 jperez999 self-assigned this Nov 29, 2022
@jperez999 jperez999 added the enhancement New feature or request label Nov 29, 2022
@github-actions
Copy link

Documentation preview

https://nvidia-merlin.github.io/systems/review/pr-237

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #237 of commit 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f, no merge conflicts.
Running as SYSTEM
Setting status of 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/536/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/237/*:refs/remotes/origin/pr/237/* # timeout=10
 > git rev-parse 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f^{commit} # timeout=10
Checking out Revision 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f # timeout=10
Commit message: "update triton to allocate more cuda memory to pool size"
 > git rev-list --no-walk 6827dfa3b38a931000abdaf33a9a695ea6224ea8 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins5502225613065573900.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+22.g8ef09f8.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.18,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.18,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+22.g8ef09f8,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='673943269'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-ywn5nv1s
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-ywn5nv1s
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=be5846c1fe1447208c4027ccf2953fb6928312afdf871f4c209cc5e5051124da
  Stored in directory: /tmp/pip-ephem-wheel-cache-ljwhpas8/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.22.0)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (0.9.0+13.g78f1f0b)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (1.1.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.8.1)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.5.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.3.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (21.3)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.10.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.5)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (7.0.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.2.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.0.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.8.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.4)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.1)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+22.g8ef09f8-py3-none-any.whl size=137437 sha256=da86e0bd81d878577b7379227448e4ad4775547a925a8d3b35a2ca195b11273e
  Stored in directory: /tmp/pip-ephem-wheel-cache-wbu8dxph/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+22.g8ef09f8
    Uninstalling merlin-systems-0.7.0+22.g8ef09f8:
      Successfully uninstalled merlin-systems-0.7.0+22.g8ef09f8
Successfully installed merlin-systems-0.7.0+22.g8ef09f8
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 103 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [ 10%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 12%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 16%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 18%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 23%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 24%]

...                                                                      [ 27%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 29%]

..                                                                       [ 31%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 34%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 35%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 41%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 42%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 46%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 71%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 74%]

tests/unit/systems/ops/implicit/test_op.py ......F..                     [ 83%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 84%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 86%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 90%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 94%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=================================== FAILURES ===================================

_______________ test_ensemble[AlternatingLeastSquares-runtime1] ________________


model_cls = <function AlternatingLeastSquares at 0x7f4c0bfb0ee0>

runtime = <merlin.systems.dag.runtimes.triton.runtime.TritonExecutorRuntime object at 0x7f4bffaa9760>

tmpdir = local('/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1')


@pytest.mark.skipif(not TRITON_SERVER_PATH, reason="triton server not found")
@pytest.mark.parametrize("runtime", [None, TritonExecutorRuntime()])
@pytest.mark.parametrize(
    "model_cls",
    [
        implicit.bpr.BayesianPersonalizedRanking,
        implicit.als.AlternatingLeastSquares,
        implicit.lmf.LogisticMatrixFactorization,
    ],
)
def test_ensemble(model_cls, runtime, tmpdir):
    model = model_cls()
    n = 100
    user_items = csr_matrix(np.random.choice([0, 1], size=n * n, p=[0.9, 0.1]).reshape(n, n))
    model.fit(user_items)

    num_to_recommend = np.random.randint(1, n)

    user_items = None
    ids, scores = model.recommend(
        [0, 1], user_items, N=num_to_recommend, filter_already_liked_items=False
    )

    implicit_op = PredictImplicit(model, num_to_recommend=num_to_recommend)

    input_schema = Schema([ColumnSchema("user_id", dtype="int64")])

    triton_chain = input_schema.column_names >> implicit_op

    triton_ens = Ensemble(triton_chain, input_schema)
    ensemble_config, _ = triton_ens.export(tmpdir, runtime=runtime)

    input_user_id = np.array([[0], [1]], dtype=np.int64)
    inputs = [
        grpcclient.InferInput(
            "user_id", input_user_id.shape, triton.np_to_triton_dtype(input_user_id.dtype)
        ),
    ]
    inputs[0].set_data_from_numpy(input_user_id)
    outputs = [grpcclient.InferRequestedOutput("scores"), grpcclient.InferRequestedOutput("ids")]

    response = None





  with run_triton_server(tmpdir) as client:





tests/unit/systems/ops/implicit/test_op.py:121:




/usr/lib/python3.8/contextlib.py:113: in enter

return next(self.gen)




model_repository = local('/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1')


@contextlib.contextmanager
def run_triton_server(
    model_repository: str,
    *,
    grpc_host: str = "localhost",
    grpc_port: int = 8001,
    backend_config: str = "tensorflow,version=2",
):
    """This function starts up a Triton server instance and returns a client to it.

    Parameters
    ----------
    model_repository : string
        The path to the model repository directory.
    grpc_host : string
        The host address for the triton gRPC server to bind to.
        Default is localhost.
    grpc_port : int
        The port for the triton gRPC server to listen on for requests.
        Default is 8001.
    backend_config : string
        A backend-specific configuration.
        Following the pattern <backend_name>,<setting>=<value>.
        Where <backend_name> is the name of the backend, such as 'tensorflow'

    Yields
    ------
    client: tritonclient.InferenceServerClient
        The client connected to the Triton server.

    """
    if grpc_port == 0 or grpc_port is None:
        grpc_port = _get_random_free_port()
    grpc_url = f"{grpc_host}:{grpc_port}"

    try:
        with grpcclient.InferenceServerClient(grpc_url) as client:
            if client.is_server_ready():
                raise RuntimeError(f"Another tritonserver is already running on {grpc_url}")
    except tritonclient.utils.InferenceServerException:
        pass

    cmdline = [
        TRITON_SERVER_PATH,
        "--model-repository",
        model_repository,
        f"--backend-config={backend_config}",
        f"--grpc-port={grpc_port}",
        f"--grpc-address={grpc_host}",
        "--cuda-memory-pool-byte-size=0:536870912",
    ]
    env = os.environ.copy()
    env["CUDA_VISIBLE_DEVICES"] = "0"
    with subprocess.Popen(cmdline, env=env) as process:
        try:
            with grpcclient.InferenceServerClient(grpc_url) as client:
                # wait until server is ready
                time_ranges = [60, 120, 300]
                for seconds in time_ranges:
                    for _ in range(seconds):
                        if process.poll() is not None:
                            retcode = process.returncode





                          raise RuntimeError(f"Tritonserver failed to start (ret={retcode})")





E                               RuntimeError: Tritonserver failed to start (ret=1)


merlin/systems/triton/utils.py:82: RuntimeError

----------------------------- Captured stderr call -----------------------------


0%|          | 0/15 [00:00<?, ?it/s]

100%|██████████| 15/15 [00:00<00:00, 2277.20it/s]

I1129 16:39:43.889746 23917 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f5606000000' with size 268435456

I1129 16:39:43.890491 23917 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 536870912

I1129 16:39:43.894091 23917 model_lifecycle.cc:459] loading: executor_model:1

1129 16:39:46.787493 23956 pb_stub.cc:241] Failed to initialize Python stub for auto-complete: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


E1129 16:39:46.816966 23917 model_lifecycle.cc:596] failed to load 'executor_model' version 1: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error


At:

/usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount

/usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup

/usr/local/lib/python3.8/dist-packages/cudf/init.py(5): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

(219): _call_with_frames_removed

(961): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load

/tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34): 

(219): _call_with_frames_removed

(848): exec_module

(686): _load_unlocked

(975): _find_and_load_unlocked

(991): _find_and_load


I1129 16:39:46.817175 23917 server.cc:561]

+------------------+------+

| Repository Agent | Path |

+------------------+------+

+------------------+------+


I1129 16:39:46.817321 23917 server.cc:588]

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Backend | Path                                                  | Config                                                                                                                                                        |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+

| python  | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} |

+---------+-------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1129 16:39:46.817547 23917 server.cc:631]

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| Model          | Version | Status                                                                                                                                                   |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+

| executor_model | 1       | UNAVAILABLE: Internal: CUDARuntimeError: cudaErrorInitializationError: initialization error                                                              |

|                |         |                                                                                                                                                          |

|                |         | At:                                                                                                                                                      |

|                |         |   /usr/local/lib/python3.8/dist-packages/rmm/_cuda/gpu.py(101): getDeviceCount                                                                           |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/utils/gpu_utils.py(55): validate_setup                                                                     |

|                |         |   /usr/local/lib/python3.8/dist-packages/cudf/init.py(5):                                                                                    |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/core/dispatch.py(51):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/dag/executors.py(22):                      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/base_runtime.py(18):  |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/runtimes/init.py(18):      |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (961): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/ensemble.py(26):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/systems/dag/init.py(20):               |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

|                |         |   /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1/executor_model/1/model.py(34):                                                 |

|                |         |   (219): _call_with_frames_removed                                                                                          |

|                |         |   (848): exec_module                                                                                               |

|                |         |   (686): _load_unlocked                                                                                                     |

|                |         |   (975): _find_and_load_unlocked                                                                                            |

|                |         |   (991): _find_and_load                                                                                                     |

+----------------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+


I1129 16:39:46.882716 23917 metrics.cc:650] Collecting metrics for GPU 0: Tesla P100-DGXS-16GB

I1129 16:39:46.883567 23917 tritonserver.cc:2214]

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| Option                           | Value                                                                                                                                                                                        |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

| server_id                        | triton                                                                                                                                                                                       |

| server_version                   | 2.25.0                                                                                                                                                                                       |

| server_extensions                | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace |

| model_repository_path[0]         | /tmp/pytest-of-jenkins/pytest-5/test_ensemble_AlternatingLeast1                                                                                                                              |

| model_control_mode               | MODE_NONE                                                                                                                                                                                    |

| strict_model_config              | 0                                                                                                                                                                                            |

| rate_limit                       | OFF                                                                                                                                                                                          |

| pinned_memory_pool_byte_size     | 268435456                                                                                                                                                                                    |

| cuda_memory_pool_byte_size{0}    | 536870912                                                                                                                                                                                    |

| response_cache_byte_size         | 0                                                                                                                                                                                            |

| min_supported_compute_capability | 6.0                                                                                                                                                                                          |

| strict_readiness                 | 1                                                                                                                                                                                            |

| exit_timeout                     | 30                                                                                                                                                                                           |

+----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


I1129 16:39:46.883603 23917 server.cc:262] Waiting for in-flight requests to complete.

I1129 16:39:46.883610 23917 server.cc:278] Timeout 30: Found 0 model versions that have in-flight inferences

I1129 16:39:46.883617 23917 server.cc:293] All models are stopped, unloading models

I1129 16:39:46.883623 23917 server.cc:300] Timeout 30: Found 0 live models and 0 in-flight non-inference requests

error: creating server: Internal - failed to load all models

W1129 16:39:47.909297 23917 metrics.cc:468] Unable to get energy consumption for GPU 0. Status:Success, value:0

=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            145      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     12    83%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2846   1223    57%


=========== 1 failed, 102 passed, 71 warnings in 1570.21s (0:26:10) ============

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   test-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins3698427695445107706.sh

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #237 of commit 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f, no merge conflicts.
Running as SYSTEM
Setting status of 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/541/ and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_systems
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/237/*:refs/remotes/origin/pr/237/* # timeout=10
 > git rev-parse 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f^{commit} # timeout=10
Checking out Revision 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f # timeout=10
Commit message: "update triton to allocate more cuda memory to pool size"
 > git rev-list --no-walk 6827dfa3b38a931000abdaf33a9a695ea6224ea8 # timeout=10
[merlin_systems] $ /bin/bash /tmp/jenkins5005938221148525611.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+22.g8ef09f8.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.19,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.19,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+22.g8ef09f8,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='1321192960'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-e58n1gy0
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-e58n1gy0
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=cbfd3fea287fdc584a91f4953aa53bed05883d31c7caad2eaefc6d031c6c81d7
  Stored in directory: /tmp/pip-ephem-wheel-cache-nl758web/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.22.0)
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (0.9.0+13.g78f1f0b)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.8.1)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.20.3)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.3.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (21.3)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.55.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.10.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.19.5)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.4.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.1)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.8.0)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.38.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (65.5.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+22.g8ef09f8-py3-none-any.whl size=139643 sha256=bfdbd24e829d0fe3af70410d9a14c5f89d2e4cdd1295a738c60532f9336d03f7
  Stored in directory: /tmp/pip-ephem-wheel-cache-r5wpfbmf/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+22.g8ef09f8
    Uninstalling merlin-systems-0.7.0+22.g8ef09f8:
      Successfully uninstalled merlin-systems-0.7.0+22.g8ef09f8
Successfully installed merlin-systems-0.7.0+22.g8ef09f8
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 103 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [ 10%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 12%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 16%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 18%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 23%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 24%]

...                                                                      [ 27%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 29%]

..                                                                       [ 31%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py FF [ 33%]

FF                                                                       [ 34%]

tests/unit/systems/ops/faiss/test_executor.py F                          [ 35%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 41%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 42%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 46%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 71%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 74%]

tests/unit/systems/ops/implicit/test_op.py .....FFFF                     [ 83%]

tests/unit/systems/ops/nvtabular/test_ensemble.py F                      [ 84%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 86%]

tests/unit/systems/ops/tf/test_ensemble.py .ERROR: Got SIGTERM, handling it as a KeyboardInterrupt

ERROR: got KeyboardInterrupt signal

Build was aborted

Aborted by �[8mha:////4I6AZwo/1Z8Fal8AhZTEatjIwqNwCcqT21311HdysuK+AAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzWEgZu/dLi1CL9xJTczDwAj6GcLcAAAAA=�[0madmin

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_systems] $ /bin/bash /tmp/jenkins4593240920231031655.sh

@jperez999
Copy link
Collaborator Author

rerun tests

3 similar comments
@jperez999
Copy link
Collaborator Author

rerun tests

@jperez999
Copy link
Collaborator Author

rerun tests

@jperez999
Copy link
Collaborator Author

rerun tests

@nvidia-merlin-bot
Copy link

Click to view CI Results
GitHub pull request #237 of commit 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f, no merge conflicts.
Running as SYSTEM
Setting status of 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_systems/546/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_systems/workspace
using credential fce1c729-5d7c-48e8-90cb-b0c314b1076e
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/systems # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/systems
 > git --version # timeout=10
using GIT_ASKPASS to set credentials login for merlin-systems user + githubtoken
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/systems +refs/pull/237/*:refs/remotes/origin/pr/237/* # timeout=10
 > git rev-parse 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f^{commit} # timeout=10
Checking out Revision 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8ef09f8c5c2e90e407ea3302fd9384a64c60eb6f # timeout=10
Commit message: "update triton to allocate more cuda memory to pool size"
 > git rev-list --no-walk 57e890261db6cbca3cbdfd3882af9f04fa123655 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins15606145788188971505.sh
PYTHONPATH=:/usr/local/lib/python3.8/dist-packages/:/usr/local/hugectr/lib:/var/jenkins_home/workspace/merlin_systems/systems
GLOB sdist-make: /var/jenkins_home/workspace/merlin_systems/systems/setup.py
test-gpu recreate: /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
test-gpu installdeps: pytest, pytest-cov
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu inst: /var/jenkins_home/workspace/merlin_systems/systems/.tox/.tmp/package/2/merlin-systems-0.7.0+22.g8ef09f8.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
test-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==22.6.0,bleach==5.0.1,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-cache==0.4.3,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.7.0+22.g8ef09f8,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
test-gpu run-test-pre: PYTHONHASHSEED='3012535524'
test-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-u68j68w0
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-u68j68w0
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 78f1f0b0952fd14b76913b0dd258565c06694abe
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+13.g78f1f0b) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+13.g78f1f0b) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (6.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (1.20.3)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+13.g78f1f0b) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+13.g78f1f0b) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+13.g78f1f0b) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+13.g78f1f0b) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+13.g78f1f0b) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+13.g78f1f0b-py3-none-any.whl size=118889 sha256=4f8d393f51e6b09220c2765beb624d247817860c1a94a2a5b05af9347f9dca87
  Stored in directory: /tmp/pip-ephem-wheel-cache-119wqw20/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+13.g78f1f0b
test-gpu run-test: commands[1] | python -m pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/merlin_systems/systems
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: nvtabular>=1.0.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (1.1.1)
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (0.9.0+13.g78f1f0b)
Requirement already satisfied: treelite-runtime==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: treelite==2.4.0 in ./.tox/test-gpu/lib/python3.8/site-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: requests<3,>=2.10 in /usr/lib/python3/dist-packages (from merlin-systems==0.7.0+22.g8ef09f8) (2.22.0)
Requirement already satisfied: numpy in /var/jenkins_home/.local/lib/python3.8/site-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.20.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from treelite==2.4.0->merlin-systems==0.7.0+22.g8ef09f8) (1.8.1)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.55.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (21.3)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (7.0.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.10.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.3.5)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.5.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.64.1)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.3.0)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.4.3)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.12.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.4.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.0.0)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (5.8.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.4)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.7.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.1.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (8.1.3)
Requirement already satisfied: tornado>=6.0.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/tornado-6.1-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.1)
Requirement already satisfied: setuptools in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/test-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->merlin-systems==0.7.0+22.g8ef09f8) (4.0.0)
Building wheels for collected packages: merlin-systems
  Building wheel for merlin-systems (pyproject.toml): started
  Building wheel for merlin-systems (pyproject.toml): finished with status 'done'
  Created wheel for merlin-systems: filename=merlin_systems-0.7.0+22.g8ef09f8-py3-none-any.whl size=139643 sha256=1c567e1ed4e79605680320d1c4bf16293a5185aa040cee4d9ab0018de1848013
  Stored in directory: /tmp/pip-ephem-wheel-cache-k1nbn_dv/wheels/20/f3/c7/f7c2f6c257e882a60cdd173901b8757359446cf94a453e0811
Successfully built merlin-systems
Installing collected packages: merlin-systems
  Attempting uninstall: merlin-systems
    Found existing installation: merlin-systems 0.7.0+22.g8ef09f8
    Uninstalling merlin-systems-0.7.0+22.g8ef09f8:
      Successfully uninstalled merlin-systems-0.7.0+22.g8ef09f8
Successfully installed merlin-systems-0.7.0+22.g8ef09f8
test-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/test-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_systems/systems, configfile: pytest.ini
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 103 items

tests/unit/test_export.py .                                              [  0%]

tests/unit/examples/test_serving_an_implicit_model_with_merlin_systems.py . [  1%]

.                                                                        [  2%]

tests/unit/examples/test_serving_an_xgboost_model_with_merlin_systems.py . [  3%]

[  3%]

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py . [  4%]

[  4%]

tests/unit/systems/dag/test_ensemble.py .                                [  5%]

tests/unit/systems/dag/test_executors.py ...                             [  8%]

tests/unit/systems/dag/test_graph.py ..                                  [ 10%]

tests/unit/systems/dag/test_model_registry.py ..                         [ 12%]

tests/unit/systems/dag/test_op_runner.py ....                            [ 16%]

tests/unit/systems/dag/ops/test_ops.py ..                                [ 18%]

tests/unit/systems/dag/runtimes/triton/test_triton.py .....              [ 23%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py . [ 24%]

...                                                                      [ 27%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py .. [ 29%]

..                                                                       [ 31%]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py .. [ 33%]

..                                                                       [ 34%]

tests/unit/systems/ops/faiss/test_executor.py .                          [ 35%]

tests/unit/systems/ops/feast/test_op.py ......                           [ 41%]

tests/unit/systems/ops/fil/test_ensemble.py .                            [ 42%]

tests/unit/systems/ops/fil/test_forest.py ....                           [ 46%]

tests/unit/systems/ops/fil/test_op.py ..........................         [ 71%]

tests/unit/systems/ops/implicit/test_executor.py ...                     [ 74%]

tests/unit/systems/ops/implicit/test_op.py .........                     [ 83%]

tests/unit/systems/ops/nvtabular/test_ensemble.py .                      [ 84%]

tests/unit/systems/ops/nvtabular/test_op.py ..                           [ 86%]

tests/unit/systems/ops/tf/test_ensemble.py ....                          [ 90%]

tests/unit/systems/ops/tf/test_op.py ....                                [ 94%]

tests/unit/systems/ops/torch/test_op.py ......                           [100%]


=============================== warnings summary ===============================

../../../../../usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33

/usr/local/lib/python3.8/dist-packages/dask_cudf/core.py:33: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

DASK_VERSION = LooseVersion(dask.version)


.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: 35 warnings

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

other = LooseVersion(other)


../../../.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18

/var/jenkins_home/.local/lib/python3.8/site-packages/nvtabular/framework_utils/init.py:18: DeprecationWarning: The nvtabular.framework_utils module is being replaced by the Merlin Models library. Support for importing from nvtabular.framework_utils is deprecated, and will be removed in a future version. Please consider using the models and layers from Merlin Models instead.

warnings.warn(


../../../../../usr/local/lib/python3.8/dist-packages/faiss/loader.py:28

/usr/local/lib/python3.8/dist-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.

if LooseVersion(numpy.version) >= "1.19":


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column x is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column y is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/test_export.py::test_export_run_ensemble_triton[tensorflow-parquet]

/var/jenkins_home/workspace/merlin_systems/systems/merlin/systems/triton/export.py:304: UserWarning: Column id is being generated by NVTabular workflow  but is unused in test_name_tf model

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

tests/unit/examples/test_serving_ranking_models_with_merlin_systems.py::test_example_04_exporting_ranking_models

/var/jenkins_home/workspace/merlin_systems/systems/.tox/test-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(


tests/unit/systems/dag/runtimes/triton/ops/fil/test_lightgbm_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py: 4 warnings

tests/unit/systems/dag/runtimes/triton/ops/fil/test_xgboost_triton.py: 4 warnings

/var/jenkins_home/.local/lib/python3.8/site-packages/pandas/core/arraylike.py:364: RuntimeWarning: invalid value encountered in log

result = getattr(ufunc, method)(*inputs, **kwargs)


tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_regressor_forest_inference[runtime1-None-executor_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime0-None-ensemble_model]

tests/unit/systems/dag/runtimes/triton/ops/fil/test_sklearn_triton.py::test_sklearn_classify_forest_inference[runtime1-None-executor_model]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_default[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_binary_classifier_with_proba[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_multi_classifier[sklearn_forest_classifier-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_regressor[sklearn_forest_regressor-get_model_params4]

tests/unit/systems/ops/fil/test_op.py::test_model_file[sklearn_forest_regressor-checkpoint.tl]

/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute n_features_ was deprecated in version 1.0 and will be removed in 1.2. Use n_features_in_ instead.

warnings.warn(msg, category=FutureWarning)


tests/unit/systems/ops/feast/test_op.py::test_feast_from_feature_view

/usr/local/lib/python3.8/dist-packages/feast/feature_view.py:100: DeprecationWarning: The argument 'input' is being deprecated. Please use 'batch_source' instead. Feast 0.13 and onwards will not support the argument 'input'.

warnings.warn(


tests/unit/systems/ops/fil/test_forest.py::test_export_merlin_models

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.


tests/unit/systems/ops/implicit/test_op.py::test_reload_from_config[AlternatingLeastSquares]

/usr/local/lib/python3.8/dist-packages/implicit/utils.py:28: UserWarning: OpenBLAS detected. Its highly recommend to set the environment variable 'export OPENBLAS_NUM_THREADS=1' to disable its internal multithreading

warnings.warn(


tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving[True-True]

tests/unit/systems/ops/torch/test_op.py::test_pytorch_op_serving_python[True-True]

/usr/local/lib/python3.8/dist-packages/torch/serialization.py:707: UserWarning: 'torch.load' received a zip file that looks like a TorchScript archive dispatching to 'torch.jit.load' (call 'torch.jit.load' directly to silence this warning)

warnings.warn("'torch.load' received a zip file that looks like a TorchScript archive"


-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html


---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                   Stmts   Miss  Cover


merlin/systems/init.py                                 6      0   100%

merlin/systems/_version.py                               354    205    42%

merlin/systems/dag/init.py                             4      0   100%

merlin/systems/dag/dictarray.py                          122     38    69%

merlin/systems/dag/ensemble.py                            46      3    93%

merlin/systems/dag/node.py                                23      1    96%

merlin/systems/dag/op_runner.py                           26      1    96%

merlin/systems/dag/ops/init.py                        11      0   100%

merlin/systems/dag/ops/compat.py                          32      8    75%

merlin/systems/dag/ops/faiss.py                           77     23    70%

merlin/systems/dag/ops/feast.py                          118      7    94%

merlin/systems/dag/ops/fil.py                            211     56    73%

merlin/systems/dag/ops/implicit.py                        72     13    82%

merlin/systems/dag/ops/operator.py                        79      6    92%

merlin/systems/dag/ops/pytorch.py                         82      5    94%

merlin/systems/dag/ops/session_filter.py                  45      7    84%

merlin/systems/dag/ops/softmax_sampling.py                51     17    67%

merlin/systems/dag/ops/tensorflow.py                      67      7    90%

merlin/systems/dag/ops/unroll_features.py                 50     50     0%

merlin/systems/dag/ops/workflow.py                        65     19    71%

merlin/systems/dag/runtimes/init.py                    1      0   100%

merlin/systems/dag/runtimes/base_runtime.py               11      2    82%

merlin/systems/dag/runtimes/triton/init.py             1      0   100%

merlin/systems/dag/runtimes/triton/ops/init.py         0      0   100%

merlin/systems/dag/runtimes/triton/ops/fil.py             95     13    86%

merlin/systems/dag/runtimes/triton/ops/operator.py        11      0   100%

merlin/systems/dag/runtimes/triton/ops/tensorflow.py      53      5    91%

merlin/systems/dag/runtimes/triton/runtime.py            145      9    94%

merlin/systems/model_registry.py                          16      1    94%

merlin/systems/triton/init.py                         49     17    65%

merlin/systems/triton/conversions.py                     143    120    16%

merlin/systems/triton/export.py                          268    155    42%

merlin/systems/triton/models/init.py                   0      0   100%

merlin/systems/triton/models/executor_model.py            35     35     0%

merlin/systems/triton/models/oprunner_model.py            33     33     0%

merlin/systems/triton/models/pytorch_model.py            105    105     0%

merlin/systems/triton/models/workflow_model.py            53     53     0%

merlin/systems/triton/utils.py                            72     14    81%

merlin/systems/workflow/init.py                       22      5    77%

merlin/systems/workflow/base.py                          113    113     0%

merlin/systems/workflow/hugectr.py                        37     37     0%

merlin/systems/workflow/pytorch.py                        10     10     0%

merlin/systems/workflow/tensorflow.py                     32     32     0%


TOTAL                                                   2846   1225    57%


================ 103 passed, 71 warnings in 1574.80s (0:26:14) =================

___________________________________ summary ____________________________________

test-gpu: commands succeeded

congratulations :)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/systems/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins9401206828193395205.sh

@jperez999 jperez999 merged commit a849a80 into NVIDIA-Merlin:main Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@karlhigley karlhigley karlhigley approved these changes

Assignees

@jperez999 jperez999

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jperez999 @nvidia-merlin-bot @karlhigley